Assessing the Severity of COVID-19 Lung Injury in Rheumatic Diseases Versus the General Population Using Deep Learning-Derived Chest Radiograph Scores.

OBJECTIVE: COVID-19 patients with rheumatic disease have a higher risk of mechanical ventilation than the general population. The present study was undertaken to assess lung involvement using a validated deep learning algorithm that extracts a quantitative measure of radiographic lung disease severity. METHODS: We performed a comparative cohort study of rheumatic disease patients with COVID-19 and ≥1 chest radiograph within ±2 weeks of COVID-19 diagnosis and matched comparators. We used unadjusted and adjusted (for age, Charlson comorbidity index, and interstitial lung disease) quantile regression to compare the maximum pulmonary x-ray severity (PXS) score at the 10th to 90th percentiles between groups. We evaluated the association of severe PXS score (>9) with mechanical ventilation and death using Cox regression. RESULTS: We identified 70 patients with rheumatic disease and 463 general population comparators. Maximum PXS scores were similar in the rheumatic disease patients and comparators at the 10th to 60th percentiles but significantly higher among rheumatic disease patients at the 70th to 90th percentiles (90th percentile score of 10.2 versus 9.2; adjusted P = 0.03). Rheumatic disease patients were more likely to have a PXS score of >9 (20% versus 11%; P = 0.02), indicating severe pulmonary disease. Rheumatic disease patients with PXS scores >9 versus ≤9 had higher risk of mechanical ventilation (hazard ratio [HR] 24.1 [95% confidence interval (95% CI) 6.7, 86.9]) and death (HR 8.2 [95% CI 0.7, 90.4]). CONCLUSION: Rheumatic disease patients with COVID-19 had more severe radiographic lung involvement than comparators. Higher PXS scores were associated with mechanical ventilation and will be important for future studies leveraging big data to assess COVID-19 outcomes in rheumatic disease patients.

Deep learning models for COVID-19 chest x-ray classification: Preventing shortcut learning using feature disentanglement.

In response to the COVID-19 global pandemic, recent research has proposed creating deep learning based models that use chest radiographs (CXRs) in a variety of clinical tasks to help manage the crisis. However, the size of existing datasets of CXRs from COVID-19+ patients are relatively small, and researchers often pool CXR data from multiple sources, for example, using different x-ray machines in various patient populations under different clinical scenarios. Deep learning models trained on such datasets have been shown to overfit to erroneous features instead of learning pulmonary characteristics in a phenomenon known as shortcut learning. We propose adding feature disentanglement to the training process. This technique forces the models to identify pulmonary features from the images and penalizes them for learning features that can discriminate between the original datasets that the images come from. We find that models trained in this way indeed have better generalization performance on unseen data; in the best case we found that it improved AUC by 0.13 on held out data. We further find that this outperforms masking out non-lung parts of the CXRs and performing histogram equalization, both of which are recently proposed methods for removing biases in CXR datasets.

Intubation and mortality prediction in hospitalized COVID-19 patients using a combination of convolutional neural network-based scoring of chest radiographs and clinical data.

Objective: To predict short-term outcomes in hospitalized COVID-19 patients using a model incorporating clinical variables with automated convolutional neural network (CNN) chest radiograph analysis. Methods: A retrospective single center study was performed on patients consecutively admitted with COVID-19 between March 14 and April 21 2020. Demographic, clinical and laboratory data were collected, and automated CNN scoring of the admission chest radiograph was performed. The two outcomes of disease progression were intubation or death within 7 days and death within 14 days following admission. Multiple imputation was performed for missing predictor variables and, for each imputed data set, a penalized logistic regression model was constructed to identify predictors and their functional relationship to each outcome. Cross-validated area under the characteristic (AUC) curves were estimated to quantify the discriminative ability of each model. Results: 801 patients (median age 59; interquartile range 46-73 years, 469 men) were evaluated. 36 patients were deceased and 207 were intubated at 7 days and 65 were deceased at 14 days. Cross-validated AUC values for predictive models were 0.82 (95% CI, 0.79-0.86) for death or intubation within 7 days and 0.82 (0.78-0.87) for death within 14 days. Automated CNN chest radiograph score was an important variable in predicting both outcomes. Conclusion: Automated CNN chest radiograph analysis, in combination with clinical variables, predicts short-term intubation and death in patients hospitalized for COVID-19 infection. Chest radiograph scoring of more severe disease was associated with a greater probability of adverse short-term outcome. Advances in knowledge: Model-based predictions of intubation and death in COVID-19 can be performed with high discriminative performance using admission clinical data and convolutional neural network-based scoring of chest radiograph severity.

Multi-population generalizability of a deep learning-based chest radiograph severity score for COVID-19.

To tune and test the generalizability of a deep learning-based model for assessment of COVID-19 lung disease severity on chest radiographs (CXRs) from different patient populations. A published convolutional Siamese neural network-based model previously trained on hospitalized patients with COVID-19 was tuned using 250 outpatient CXRs. This model produces a quantitative measure of COVID-19 lung disease severity (pulmonary x-ray severity (PXS) score). The model was evaluated on CXRs from 4 test sets, including 3 from the United States (patients hospitalized at an academic medical center (N = 154), patients hospitalized at a community hospital (N = 113), and outpatients (N = 108)) and 1 from Brazil (patients at an academic medical center emergency department (N = 303)). Radiologists from both countries independently assigned reference standard CXR severity scores, which were correlated with the PXS scores as a measure of model performance (Pearson R). The Uniform Manifold Approximation and Projection (UMAP) technique was used to visualize the neural network results. Tuning the deep learning model with outpatient data showed high model performance in 2 United States hospitalized patient datasets (R = 0.88 and R = 0.90, compared to baseline R = 0.86). Model performance was similar, though slightly lower, when tested on the United States outpatient and Brazil emergency department datasets (R = 0.86 and R = 0.85, respectively). UMAP showed that the model learned disease severity information that generalized across test sets. A deep learning model that extracts a COVID-19 severity score on CXRs showed generalizable performance across multiple populations from 2 continents, including outpatients and hospitalized patients.

External COVID-19 Deep Learning Model Validation on ACR AI-LAB: It's a Brave New World.

PURPOSE: Deploying external artificial intelligence (AI) models locally can be logistically challenging. We aimed to use the ACR AI-LAB software platform for local testing of a chest radiograph (CXR) algorithm for COVID-19 lung disease severity assessment. METHODS: An externally developed deep learning model for COVID-19 radiographic lung disease severity assessment was loaded into the AI-LAB platform at an independent academic medical center, which was separate from the institution in which the model was trained. The data set consisted of CXR images from 141 patients with reverse transcription-polymerase chain reaction-confirmed COVID-19, which were routed to AI-LAB for model inference. The model calculated a Pulmonary X-ray Severity (PXS) score for each image. This score was correlated with the average of a radiologist-based assessment of severity, the modified Radiographic Assessment of Lung Edema score, independently interpreted by three radiologists. The associations between the PXS score and patient admission and intubation or death were assessed. RESULTS: The PXS score deployed in AI-LAB correlated with the radiologist-determined modified Radiographic Assessment of Lung Edema score (r = 0.80). PXS score was significantly higher in patients who were admitted (4.0 versus 1.3, P < .001) or intubated or died within 3 days (5.5 versus 3.3, P = .001). CONCLUSIONS: AI-LAB was successfully used to test an external COVID-19 CXR AI algorithm on local data with relative ease, showing generalizability of the PXS score model. For AI models to scale and be clinically useful, software tools that facilitate the local testing process, like the freely available AI-LAB, will be important to cross the AI implementation gap in health care systems.

Radiology Implementation Considerations for Artificial Intelligence (AI) Applied to COVID-19, From the AJR Special Series on AI Applications.

Hundreds of imaging-based artificial intelligence (AI) models have been developed in response to the COVID-19 pandemic. AI systems that incorporate imaging have shown promise in primary detection, severity grading, and prognostication of outcomes in COVID-19, and have enabled integration of imaging with a broad range of additional clinical and epidemiologic data. However, systematic reviews of AI models applied to COVID-19 medical imaging have highlighted problems in the field, including methodologic issues and problems in real-world deployment. Clinical use of such models should be informed by both the promise and potential pitfalls of implementation. How does a practicing radiologist make sense of this complex topic, and what factors should be considered in the implementation of AI tools for imaging of COVID-19? This critical review aims to help the radiologist understand the nuances that impact the clinical deployment of AI for imaging of COVID-19. We review imaging use cases for AI models in COVID-19 (e.g., diagnosis, severity assessment, and prognostication) and explore considerations for AI model development and testing, deployment infrastructure, clinical user interfaces, quality control, and institutional review board and regulatory approvals, with a practical focus on what a radiologist should consider when implementing an AI tool for COVID-19.

Automated tracking of emergency department abdominal CT findings during the COVID-19 pandemic using natural language processing.

PURPOSE: During the COVID-19 pandemic, emergency department (ED) volumes have fluctuated. We hypothesized that natural language processing (NLP) models could quantify changes in detection of acute abdominal pathology (acute appendicitis (AA), acute diverticulitis (AD), or bowel obstruction (BO)) on CT reports. METHODS: This retrospective study included 22,182 radiology reports from CT abdomen/pelvis studies performed at an urban ED between January 1, 2018 to August 14, 2020. Using a subset of 2448 manually annotated reports, we trained random forest NLP models to classify the presence of AA, AD, and BO in report impressions. Performance was assessed using 5-fold cross validation. The NLP classifiers were then applied to all reports. RESULTS: The NLP classifiers for AA, AD, and BO demonstrated cross-validation classification accuracies between 0.97 and 0.99 and F1-scores between 0.86 and 0.91. When applied to all CT reports, the estimated numbers of AA, AD, and BO cases decreased 43-57% in April 2020 (first regional peak of COVID-19 cases) compared to 2018-2019. However, the number of abdominal pathologies detected rebounded in May-July 2020, with increases above historical averages for AD. The proportions of CT studies with these pathologies did not significantly increase during the pandemic period. CONCLUSION: Dramatic decreases in numbers of acute abdominal pathologies detected by ED CT studies were observed early on during the COVID-19 pandemic, though these numbers rapidly rebounded. The proportions of CT cases with these pathologies did not increase, which suggests patients deferred care during the first pandemic peak. NLP can help automatically track findings in ED radiology reporting.

Prolonged Intubation in Patients With Prior Cerebrovascular Disease and COVID-19.

Objectives: Patients with comorbidities are at increased risk for poor outcomes in COVID-19, yet data on patients with prior neurological disease remains limited. Our objective was to determine the odds of critical illness and duration of mechanical ventilation in patients with prior cerebrovascular disease and COVID-19. Methods: A observational study of 1,128 consecutive adult patients admitted to an academic center in Boston, Massachusetts, and diagnosed with laboratory-confirmed COVID-19. We tested the association between prior cerebrovascular disease and critical illness, defined as mechanical ventilation (MV) or death by day 28, using logistic regression with inverse probability weighting of the propensity score. Among intubated patients, we estimated the cumulative incidence of successful extubation without death over 45 days using competing risk analysis. Results: Of the 1,128 adults with COVID-19, 350 (36%) were critically ill by day 28. The median age of patients was 59 years (SD: 18 years) and 640 (57%) were men. As of June 2nd, 2020, 127 (11%) patients had died. A total of 177 patients (16%) had a prior cerebrovascular disease. Prior cerebrovascular disease was significantly associated with critical illness (OR = 1.54, 95% CI = 1.14-2.07), lower rate of successful extubation (cause-specific HR = 0.57, 95% CI = 0.33-0.98), and increased duration of intubation (restricted mean time difference = 4.02 days, 95% CI = 0.34-10.92) compared to patients without cerebrovascular disease. Interpretation: Prior cerebrovascular disease adversely affects COVID-19 outcomes in hospitalized patients. Further study is required to determine if this subpopulation requires closer monitoring for disease progression during COVID-19.

Right Ventricular Strain Is Common in Intubated COVID-19 Patients and Does Not Reflect Severity of Respiratory Illness.

BACKGROUND: Right ventricular (RV) dysfunction is common and associated with worse outcomes in patients with coronavirus disease 2019 (COVID-19). In non-COVID-19 acute respiratory distress syndrome, RV dysfunction develops due to pulmonary hypoxic vasoconstriction, inflammation, and alveolar overdistension or atelectasis. Although similar pathogenic mechanisms may induce RV dysfunction in COVID-19, other COVID-19-specific pathology, such as pulmonary endothelialitis, thrombosis, or myocarditis, may also affect RV function. We quantified RV dysfunction by echocardiographic strain analysis and investigated its correlation with disease severity, ventilatory parameters, biomarkers, and imaging findings in critically ill COVID-19 patients. METHODS: We determined RV free wall longitudinal strain (FWLS) in 32 patients receiving mechanical ventilation for COVID-19-associated respiratory failure. Demographics, comorbid conditions, ventilatory parameters, medications, and laboratory findings were extracted from the medical record. Chest imaging was assessed to determine the severity of lung disease and the presence of pulmonary embolism. RESULTS: Abnormal FWLS was present in 66% of mechanically ventilated COVID-19 patients and was associated with higher lung compliance (39.6 vs 29.4 mL/cmH2O, P = 0.016), lower airway plateau pressures (21 vs 24 cmH2O, P = 0.043), lower tidal volume ventilation (5.74 vs 6.17 cc/kg, P = 0.031), and reduced left ventricular function. FWLS correlated negatively with age (r = -0.414, P = 0.018) and with serum troponin (r = 0.402, P = 0.034). Patients with abnormal RV strain did not exhibit decreased oxygenation or increased disease severity based on inflammatory markers, vasopressor requirements, or chest imaging findings. CONCLUSIONS: RV dysfunction is common among critically ill COVID-19 patients and is not related to abnormal lung mechanics or ventilatory pressures. Instead, patients with abnormal FWLS had more favorable lung compliance. RV dysfunction may be secondary to diffuse intravascular micro- and macro-thrombosis or direct myocardial damage. TRIAL REGISTRATION: National Institutes of Health #NCT04306393. Registered 10 March 2020, https://clinicaltrials.gov/ct2/show/NCT04306393.

The RSNA International COVID-19 Open Radiology Database (RICORD).

The coronavirus disease 2019 (COVID-19) pandemic is a global health care emergency. Although reverse-transcription polymerase chain reaction testing is the reference standard method to identify patients with COVID-19 infection, chest radiography and CT play a vital role in the detection and management of these patients. Prediction models for COVID-19 imaging are rapidly being developed to support medical decision making. However, inadequate availability of a diverse annotated data set has limited the performance and generalizability of existing models. To address this unmet need, the RSNA and Society of Thoracic Radiology collaborated to develop the RSNA International COVID-19 Open Radiology Database (RICORD). This database is the first multi-institutional, multinational, expert-annotated COVID-19 imaging data set. It is made freely available to the machine learning community as a research and educational resource for COVID-19 chest imaging. Pixel-level volumetric segmentation with clinical annotations was performed by thoracic radiology subspecialists for all COVID-19-positive thoracic CT scans. The labeling schema was coordinated with other international consensus panels and COVID-19 data annotation efforts, the European Society of Medical Imaging Informatics, the American College of Radiology, and the American Association of Physicists in Medicine. Study-level COVID-19 classification labels for chest radiographs were annotated by three radiologists, with majority vote adjudication by board-certified radiologists. RICORD consists of 240 thoracic CT scans and 1000 chest radiographs contributed from four international sites. It is anticipated that RICORD will ideally lead to prediction models that can demonstrate sustained performance across populations and health care systems.

CoVA: An Acuity Score for Outpatient Screening that Predicts Coronavirus Disease 2019 Prognosis.

BACKGROUND: We sought to develop an automatable score to predict hospitalization, critical illness, or death for patients at risk for coronavirus disease 2019 (COVID-19) presenting for urgent care. METHODS: We developed the COVID-19 Acuity Score (CoVA) based on a single-center study of adult outpatients seen in respiratory illness clinics or the emergency department. Data were extracted from the Partners Enterprise Data Warehouse, and split into development (n = 9381, 7 March-2 May) and prospective (n = 2205, 3-14 May) cohorts. Outcomes were hospitalization, critical illness (intensive care unit or ventilation), or death within 7 days. Calibration was assessed using the expected-to-observed event ratio (E/O). Discrimination was assessed by area under the receiver operating curve (AUC). RESULTS: In the prospective cohort, 26.1%, 6.3%, and 0.5% of patients experienced hospitalization, critical illness, or death, respectively. CoVA showed excellent performance in prospective validation for hospitalization (expected-to-observed ratio [E/O]: 1.01; AUC: 0.76), for critical illness (E/O: 1.03; AUC: 0.79), and for death (E/O: 1.63; AUC: 0.93). Among 30 predictors, the top 5 were age, diastolic blood pressure, blood oxygen saturation, COVID-19 testing status, and respiratory rate. CONCLUSIONS: CoVA is a prospectively validated automatable score for the outpatient setting to predict adverse events related to COVID-19 infection.

Multi-Radiologist User Study for Artificial Intelligence-Guided Grading of COVID-19 Lung Disease Severity on Chest Radiographs.

RATIONALE AND OBJECTIVES: Radiographic findings of COVID-19 pneumonia can be used for patient risk stratification; however, radiologist reporting of disease severity is inconsistent on chest radiographs (CXRs). We aimed to see if an artificial intelligence (AI) system could help improve radiologist interrater agreement. MATERIALS AND METHODS: We performed a retrospective multi-radiologist user study to evaluate the impact of an AI system, the PXS score model, on the grading of categorical COVID-19 lung disease severity on 154 chest radiographs into four ordinal grades (normal/minimal, mild, moderate, and severe). Four radiologists (two thoracic and two emergency radiologists) independently interpreted 154 CXRs from 154 unique patients with COVID-19 hospitalized at a large academic center, before and after using the AI system (median washout time interval was 16 days). Three different thoracic radiologists assessed the same 154 CXRs using an updated version of the AI system trained on more imaging data. Radiologist interrater agreement was evaluated using Cohen and Fleiss kappa where appropriate. The lung disease severity categories were associated with clinical outcomes using a previously published outcomes dataset using Fisher's exact test and Chi-square test for trend. RESULTS: Use of the AI system improved radiologist interrater agreement (Fleiss κ = 0.40 to 0.66, before and after use of the system). The Fleiss κ for three radiologists using the updated AI system was 0.74. Severity categories were significantly associated with subsequent intubation or death within 3 days. CONCLUSION: An AI system used at the time of CXR study interpretation can improve the interrater agreement of radiologists.

Improvement and Multi-Population Generalizability of a Deep Learning-Based Chest Radiograph Severity Score for COVID-19.

PURPOSE: To improve and test the generalizability of a deep learning-based model for assessment of COVID-19 lung disease severity on chest radiographs (CXRs) from different patient populations. MATERIALS AND METHODS: A published convolutional Siamese neural network-based model previously trained on hospitalized patients with COVID-19 was tuned using 250 outpatient CXRs. This model produces a quantitative measure of COVID-19 lung disease severity (pulmonary x-ray severity (PXS) score). The model was evaluated on CXRs from four test sets, including 3 from the United States (patients hospitalized at an academic medical center (N=154), patients hospitalized at a community hospital (N=113), and outpatients (N=108)) and 1 from Brazil (patients at an academic medical center emergency department (N=303)). Radiologists from both countries independently assigned reference standard CXR severity scores, which were correlated with the PXS scores as a measure of model performance (Pearson r). The Uniform Manifold Approximation and Projection (UMAP) technique was used to visualize the neural network results. RESULTS: Tuning the deep learning model with outpatient data improved model performance in two United States hospitalized patient datasets (r=0.88 and r=0.90, compared to baseline r=0.86). Model performance was similar, though slightly lower, when tested on the United States outpatient and Brazil emergency department datasets (r=0.86 and r=0.85, respectively). UMAP showed that the model learned disease severity information that generalized across test sets. CONCLUSIONS: Performance of a deep learning-based model that extracts a COVID-19 severity score on CXRs improved using training data from a different patient cohort (outpatient versus hospitalized) and generalized across multiple populations.

Automated Assessment and Tracking of COVID-19 Pulmonary Disease Severity on Chest Radiographs using Convolutional Siamese Neural Networks.

PURPOSE: To develop an automated measure of COVID-19 pulmonary disease severity on chest radiographs (CXRs), for longitudinal disease tracking and outcome prediction. MATERIALS AND METHODS: A convolutional Siamese neural network-based algorithm was trained to output a measure of pulmonary disease severity on CXRs (pulmonary x-ray severity (PXS) score), using weakly-supervised pretraining on ∼160,000 anterior-posterior images from CheXpert and transfer learning on 314 frontal CXRs from COVID-19 patients. The algorithm was evaluated on internal and external test sets from different hospitals (154 and 113 CXRs respectively). PXS scores were correlated with radiographic severity scores independently assigned by two thoracic radiologists and one in-training radiologist (Pearson r). For 92 internal test set patients with follow-up CXRs, PXS score change was compared to radiologist assessments of change (Spearman ρ). The association between PXS score and subsequent intubation or death was assessed. Bootstrap 95% confidence intervals (CI) were calculated. RESULTS: PXS scores correlated with radiographic pulmonary disease severity scores assigned to CXRs in the internal and external test sets (r=0.86 (95%CI 0.80-0.90) and r=0.86 (95%CI 0.79-0.90) respectively). The direction of change in PXS score in follow-up CXRs agreed with radiologist assessment (ρ=0.74 (95%CI 0.63-0.81)). In patients not intubated on the admission CXR, the PXS score predicted subsequent intubation or death within three days of hospital admission (area under the receiver operating characteristic curve=0.80 (95%CI 0.75-0.85)). CONCLUSION: A Siamese neural network-based severity score automatically measures radiographic COVID-19 pulmonary disease severity, which can be used to track disease change and predict subsequent intubation or death.